VLCDoC: Vision-Language contrastive pre-training model for cross-Modal document classification

Multimodal learning from document data has achieved great success lately as it allows to pre-train semantically meaningful features a prior into learnable downstream task. In this paper, we approach the classification problem by cross-modal representations through language and vision cues, considering intra- inter-modality relationships. Instead of merging different modalities joint representation space, proposed method exploits high-level interactions learns relevant semantic information effective attention flows within across modalities. The objective is devised between alignment tasks, where similarity distribution per task computed contracting positive sample pairs while simultaneously contrasting negative ones in space. Extensive experiments on public benchmark datasets demonstrate effectiveness generality our model both low-scale large-scale datasets.